---
title: Scalability & Latency
layout: default
pageOrder: 6
section: 'General'
subsection: true
sitemap:
priority: 0.8
changefreq: 'monthly'
lastmod: 2019-11-10T08:00:00+01:00
---

<div class="header">
    <h1>Scalability & Latency</h1>

    <h2 itemprop="headline">average of 1.58ms and p99 of 4ms for 150 parallel clients sending 95,228 requests per second</h2>
</div>

<p>MockServer is build to support massive scale from a single instance:</p>
<ul>
    <li><a href="#apache_bench">Apache Benchmark</a> tested up to <strong>6,000 parallel clients</strong> and shows MockServer has an <strong>average of 1.58ms and p99 of 4ms for 150 parallel clients sending 95,228 requests per second</strong></li>
    <li><a href="#locust">Locust</a> tested up to <strong>3,000 parallel clients</strong> and shows MockServer has a <strong>p99 of 4ms for 150 parallel clients</strong></li>
</ul>

<p>Both performance testing frameworks show similar results up to 2,000 parallel clients at which point Locust reports warnings and the figures are no longer consistent with Apache Benchmark.</p>

<p>The following frameworks & techniques are used to maximise scalability:</p>
<ul>
    <li><a href="https://netty.io">Netty</a> an asynchronous event-driven network application framework to maximise the scalability of HTTP and TLS</li>
    <li><a href="https://lmax-exchange.github.io/disruptor">LMAX Disruptor</a> a high performance inter-thread messaging library to maximise the scalability of recording events (i.e. state) and logging</li>
    <li><a href="https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ScheduledThreadPoolExecutor.html">ScheduledThreadPoolExecutor</a> a thread pool that can scheduled delayed tasks is used to execute delay response to avoid blocking threads</li>
</ul>

<h2>Performance Tests</h2>

<p>MockServer has been performance tested using <a href="#apache_bench">Apache Benchmark</a> and <a href="#locust">Locust</a> with the following scenario:</p>
<ul>
    <li>four basic expectations, including method, path and headers</li>
    <li>basic GET request matching third expectation (i.e. three matches are attempted for each request)</li>
</ul>

<p>During the test MockServer was run on a Java 13 JVM, with the following command:</p>

<pre class="prettyprint lang-bash code"><code class="code">java -Xmx500m -Dmockserver.logLevel=WARN -Dmockserver.disableSystemOut=true -jar ~/.m2/repository/org/mock-server/mockserver-netty/5.11.2-SNAPSHOT/mockserver-netty-5.11.2-SNAPSHOT-jar-with-dependencies.jar -serverPort 1080</code></pre>

<p>The following graph shows how the p99 increases as the number of parallel clients increase.</p>

<img class="ui_image" src="/images/p99vsparallelclients.png" alt="p99 as parallel clients increase">

<a id="apache_bench" class="anchor" href="#apache_bench">&nbsp;</a>

<h3>Apache Bench Performance Test Results</h3>

<p><a href="https://httpd.apache.org/docs/2.4/programs/ab.html">Apache Benchmark</a> was executed as follows:</p>

<pre class="prettyprint lang-bash code"><code class="code">ab -k -n 10000000 -c &lt;parallel clients&gt; http://127.0.0.1:1080/simple</code></pre>

<p>The test results are:</p>

<table class="results">
    <thead>
    <tr>
        <td>parallel clients</td>
        <td>50%</td>
        <td>66%</td>
        <td>75%</td>
        <td>80%</td>
        <td>90%</td>
        <td>95%</td>
        <td>98%</td>
        <td>99%</td>
        <td>requests/s</td>
        <td>mean</td>
    </tr>
    </thead>
    <tbody>
    <tr>
        <td>10</td>
        <td>0</td>
        <td>0</td>
        <td>0</td>
        <td>0</td>
        <td>0</td>
        <td>0</td>
        <td>0</td>
        <td>0</td>
        <td>77,122</td>
        <td>0.13</td>
    </tr>
    <tr>
        <td>50</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>85,765</td>
        <td>0.58</td>
    </tr>
    <tr>
        <td>100</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>2</td>
        <td>2</td>
        <td>3</td>
        <td>92,846</td>
        <td>1.08</td>
    </tr>
    <tr>
        <td>150</td>
        <td>1</td>
        <td>2</td>
        <td>2</td>
        <td>2</td>
        <td>2</td>
        <td>2</td>
        <td>3</td>
        <td>4</td>
        <td>95,228</td>
        <td>1.58</td>
    </tr>
    <tr>
        <td>250</td>
        <td>3</td>
        <td>3</td>
        <td>3</td>
        <td>3</td>
        <td>4</td>
        <td>5</td>
        <td>7</td>
        <td>8</td>
        <td>86,470</td>
        <td>2.89</td>
    </tr>
    <tr>
        <td>500</td>
        <td>6</td>
        <td>6</td>
        <td>6</td>
        <td>6</td>
        <td>7</td>
        <td>7</td>
        <td>8</td>
        <td>9</td>
        <td>83,209</td>
        <td>6.01</td>
    </tr>
    <tr>
        <td>750</td>
        <td>9</td>
        <td>9</td>
        <td>10</td>
        <td>10</td>
        <td>11</td>
        <td>11</td>
        <td>12</td>
        <td>15</td>
        <td>75,554</td>
        <td>9.93</td>
    </tr>
    <tr>
        <td>1000</td>
        <td>11</td>
        <td>12</td>
        <td>13</td>
        <td>13</td>
        <td>14</td>
        <td>16</td>
        <td>17</td>
        <td>21</td>
        <td>75,423</td>
        <td>13.26</td>
    </tr>
    <tr>
        <td>2000</td>
        <td>24</td>
        <td>24</td>
        <td>25</td>
        <td>26</td>
        <td>27</td>
        <td>29</td>
        <td>31</td>
        <td>35</td>
        <td>82,191</td>
        <td>24.33</td>
    </tr>
    <tr>
        <td>3000</td>
        <td>37</td>
        <td>39</td>
        <td>40</td>
        <td>40</td>
        <td>43</td>
        <td>46</td>
        <td>51</td>
        <td>58</td>
        <td>78,171</td>
        <td>38.38</td>
        <td></td>
    </tr>
    <tr>
        <td>4000</td>
        <td>52</td>
        <td>55</td>
        <td>57</td>
        <td>59</td>
        <td>64</td>
        <td>70</td>
        <td>82</td>
        <td>91</td>
        <td>73,552</td>
        <td>54.38</td>
    </tr>
    <tr>
        <td>5000</td>
        <td>65</td>
        <td>67</td>
        <td>70</td>
        <td>71</td>
        <td>75</td>
        <td>79</td>
        <td>90</td>
        <td>102</td>
        <td>74,065</td>
        <td>67.51</td>
    </tr>
    <tr>
        <td>6000</td>
        <td>80</td>
        <td>84</td>
        <td>88</td>
        <td>90</td>
        <td>97</td>
        <td>104</td>
        <td>122</td>
        <td>137</td>
        <td>70,432</td>
        <td>85.19</td>
    </tr>
    </tbody>
</table>

<a id="locust" class="anchor" href="#locust">&nbsp;</a>

<h3>Locust Performance Test Results</h3>

<p><a href="https://httpd.apache.org/docs/2.4/programs/ab.html">Apache Benchmark</a> was executed as follows:</p>

<pre class="prettyprint lang-bash code"><code class="code">locust --loglevel=WARNING --headless --only-summary -u &lt;parallel clients&gt; -r 100 -t 180 --host=http://127.0.0.1:1080</code></pre>

<p>The test results are:</p>

<table class="results">
    <thead>
    <tr>
        <td>parallel clients</td>
        <td>50%</td>
        <td>66%</td>
        <td>75%</td>
        <td>80%</td>
        <td>90%</td>
        <td>95%</td>
        <td>98%</td>
        <td>99%</td>
        <td>99.90%</td>
        <td>99.99%</td>
        <td>requests/s</td>
        <td>mean</td>
    </tr>
    </thead>
    <tbody>
    <tr>
        <td>10</td>
        <td>0</td>
        <td>0</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>2</td>
        <td>5</td>
        <td>5</td>
        <td>11</td>
        <td>0</td>
    </tr>
    <tr>
        <td>50</td>
        <td>0</td>
        <td>0</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>2</td>
        <td>3</td>
        <td>5</td>
        <td>50</td>
        <td>0</td>
    </tr>
    <tr>
        <td>100</td>
        <td>0</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>2</td>
        <td>2</td>
        <td>3</td>
        <td>4</td>
        <td>8</td>
        <td>100</td>
        <td>0</td>
    </tr>
    <tr>
        <td>150</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>1</td>
        <td>2</td>
        <td>3</td>
        <td>3</td>
        <td>4</td>
        <td>5</td>
        <td>6</td>
        <td>149</td>
        <td>0</td>
    </tr>
    <tr>
        <td>250</td>
        <td>2</td>
        <td>3</td>
        <td>3</td>
        <td>4</td>
        <td>5</td>
        <td>6</td>
        <td>7</td>
        <td>8</td>
        <td>15</td>
        <td>46</td>
        <td>245</td>
        <td>2</td>
    </tr>
    <tr>
        <td>500</td>
        <td>2</td>
        <td>2</td>
        <td>3</td>
        <td>3</td>
        <td>4</td>
        <td>5</td>
        <td>6</td>
        <td>7</td>
        <td>9</td>
        <td>46</td>
        <td>479</td>
        <td>2</td>
    </tr>
    <tr>
        <td>750</td>
        <td>3</td>
        <td>4</td>
        <td>5</td>
        <td>6</td>
        <td>8</td>
        <td>10</td>
        <td>12</td>
        <td>14</td>
        <td>29</td>
        <td>34</td>
        <td>699</td>
        <td>3</td>
    </tr>
    <tr>
        <td>1000</td>
        <td>3</td>
        <td>4</td>
        <td>6</td>
        <td>6</td>
        <td>8</td>
        <td>10</td>
        <td>13</td>
        <td>16</td>
        <td>36</td>
        <td>52</td>
        <td>909</td>
        <td>3</td>
    </tr>
    <tr>
        <td>2000</td>
        <td>4</td>
        <td>7</td>
        <td>10</td>
        <td>12</td>
        <td>22</td>
        <td>34</td>
        <td>49</td>
        <td>59</td>
        <td>87</td>
        <td>110</td>
        <td>1626.14</td>
        <td>8</td>
    </tr>
    <tr>
        <td>3000</td>
        <td>51</td>
        <td>78</td>
        <td>99</td>
        <td>110</td>
        <td>160</td>
        <td>180</td>
        <td>220</td>
        <td>240</td>
        <td>290</td>
        <td>310</td>
        <td>2629.92</td>
        <td>54</td>
    </tr>
    </tbody>
</table>

<p>The following <strong>locustfile.py</strong> was used</p>

<pre class="prettyprint lang-python code"><code class="code">import locust.stats
from locust import task, between
locust.stats.CONSOLE_STATS_INTERVAL_SEC = 60
from locust.contrib.fasthttp import FastHttpLocust

class UserBehavior(FastHttpUser):
    wait_time = between(1, 1)

    @task
    def request(self):
        self.client.get("/simple", verify=False)
</code></pre>

{% include_subpage _includes/clustering.html %}

{% include_subpage _includes/performance_configuration.html %}
